042390. P 10587 



APPLICATION FOR UNITED STATES LETTERS PATENT 



FOR 

METHOD OF PERFORMING VIDEO ENCODING RATE CONTROL USING BIT BUDGET 



Inventor(s): Hyun Mun Kim 
Tinku Acharya 



Prepared by: Howard Skaist, 

Senior IP Attorney 

Intel Corporation 

21 1 1 N. E. 25th Avenue; JF3-147 
Hillsboro, OR 97124 
Phone: (503)264-0967 
Facsimile: (503)264-1729 



ELD3MM371DMUS 

t 



1 



I 

RM1N( 



042390.P10587 

METHOD OF PERFOR«G VIDEO ENCODING RATE CONTROL USING BIT BUDGET 




RELATED APPLICATIONS 



This patent application is related to concurrently filed US Patent Application Serial No. 

, titled "Method of Performing Video Encoding Rate Control", by Kim et al. 

(attorney docket no. 042390.P10264), filed on , and to concurrently 

filed US Patent Application Serial No. , titled "Method of Performing Video 

Encoding Rate Control Using Motion Estimation", by Kim et al. (attorney docket no. 

042390.p10265), filed on , both assigned to the assignee of the 

present invention and herein incorporated by reference. 



BACKGROUND 



The present disclosure is related to rate control of the encoding of video images. 



As is well-known, video encoding may be performed by any one of a number or variety 
of techniques. Common techniques that are frequently employed comply with certain 
established standards, such as the 'MPEG' (Moving Pictures Expert Group) and 'H.26x' 
standards. These include the following: ITU-T "Video coding for low bit-rate communications," 
ITU-T Recommendation H.263, version 1, Nov. 1995 and version 2, Jan. 1998; "Generic 
Coding of Moving Pictures and Associated Audio Information: Video," ISO/IEC 13818-2: 
International Standard 1995; and "Coding of audio-visual Objects-Part 2: Visual Amendment 1 ; 
Visual extensions," ISO/IEC 14496-2: Draft of January 6, 2000; respectively, referred to 
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Specifically as H.263, H.263+, MPEG-2, and MPEG-4 and generally as MPEG and H.26x, 
hereinafter. Such standards, however, define bit stream syntax so that any standard 
compliant decoder may be employed to decode the encoded video. This provides encoders 
with a relatively large amount of flexibility in terms of implementation. 

Rate control, such as bit rate control, is one of the issues not generally specified for the 
video encoder, therefore, making it possible to provide the capability to employ a variety of 
different techniques. Furthermore, applying rate control may impact the processing of video in 
several respects. In one respect, rate control may be employed to maintain buffer constraints 
and, thereby, prevent overflow and/or underflow during encoding and, in particular, in 
connection with real-time applications. Likewise, in another respect, rate control may also 
impact picture quality. 

In addition to rate control, there are other parameters that may be varied by an encoder 
during encoding that may result in various levels of image distortion, and, therefore, impact 
performance. Therefore, it may, at times, be difficult to improve the image quality to an 
acceptable level while also meeting various desired constraints, such as, for example, a total 
bit budget, employing an appropriate amount of delay, etc. 

Techniques exist to balance these considerations in order to provide acceptable or 
desirable solutions. Examples include employing Lagrangian optimization or dynamic 
programming. Unfortunately, however, such techniques are frequently or typically 
computationally complex and, therefore, expensive in terms of the amount of processing 
resources consumed by such an approach. Furthermore, the dependency that typically exists 
between images or image frames in the processing of video may, at times, make addressing 
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such issues even more complex. For example, the distortion of the current frame may depend 
at least in part on the selection of quantization parameters, for example, for the previous frame 
or frames. Therefore, a technique for performing rate control in video encoding that is less 
computationally complex than previous or state of the art approaches, but that also balances 
at least some of the foregoing complex considerations, is desirable. 



BRIEF DESCRIPTION OF THE DRAWINGS 



The subject matter regarded as the invention as particularly pointed out and distinctly 
claimed in the concluding portion of the specification. The invention, however, both as to 
organization and method of operation, together with objects, features, and advantages thereof, 
may best be understood by reference to the following detailed description when read with the 
accompanying drawings in which: 



FIG. 1 is block diagram illustrating a video encoder that may employ an embodiment of 
a method of performing video encoding rate control using bit budget in accordance with the 
present invention; 



FIG. 2 is a series of plots illustrating the relationship between macroblock SAD and bit 
count for various picture types for a specific number of quantization bins; and 



FIG. 3 is a table illustrating the comparison of performance parameters between the 02 
control approach and an embodiment of a method of performing video encoding rate control 
using bit budget in accordance with the present invention. 



DETAILED DESCRIPTIO 
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In the following detailed description, numerous specific details are set forth in order to 
provide a thorough understanding of the invention. However, it will be understood by those 
skilled in the art that the present invention may be practiced without these specific details. In 
other instances, well-known methods, procedures, components and circuits have not been 
described in detail so as not to obscure the present invention. 



As previously described, video encoding rate control may be a feature of a video 
encoder. Although the invention is not limited in scope in this respect, in one embodiment of a 
method of performing video encoding rate control using bit budget in accordance with the 
present invention, the video bit rate employed during video encoding is varied based at least in 
part on a measurement of the variation in pixel signal level values for a selected portion of a 
video image being encoded. One rationale for employing such an approach may be that the 
larger the range or variation in pixel signal level values, the greater the number of bits 
employed to represent or differentiate the different pixel signal value levels and vice versa. 



Therefore, for this particular embodiment, although, of course, the invention is not 
limited in scope in this respect, a relationship may be employed between the variation in pixel 
signal value levels for a macroblock and the associated video encoding rate control to be 
applied by the video encoder. Furthermore, although there are a variety of techniques that 
may be employed to measure variation and the invention is not limited in scope to any 
particular technique, in this particular embodiment, the variation may be measured using the 
sum of absolute differences (SAD), which is employed in motion estimation. 
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SAD =minY,Y \C[i,j]-R[x^+x + i,y^ + y + j]\ 



[1] 



;=0 i=0 



where 



(^0' Yo) 
C/x, y] 

m y] 



upper left corner coordinates of the current macroblock 
current macroblock luminance samples 
reconstructed previous frame luminance samples 
search range: {(jc, y) :- 16 <x,y<l6 ] 



S 



As is well-known, the Sy^D values are computed in all or some selected search points in the search 
space (S). The motion vector {MVx, MVy) is selected based on the displacement of the search point 
which results in the minimum SAD among all the SAD values in the search space. It is noted, of 
course, that other potential measures of variation are essentially a substitute of the SAD. For 
example, the mean absolute difference (MAD) may be employed in place of the SAD and 
should provide nearly identical, if not identical, results. Therefore, such other substitute 
measures are clearly within the scope of the invention. 

In this context, the SAD provides several advantages. It is already computed as part of 
motion estimation, and, therefore, introduces little or no additional overhead in terms of the 
consumption of processing resources. Furthermore, motion estimation provides information 
that may be useful in terms of video encoding rate control. For example, motion estimation 
provides information about prediction mode decisions, motion vector choices, and displaced 
frame difference coding fidelity. 



In this context, it is noted that a modification in quantization step size specifically results 
in an adjustment of a video encoding rate, here the video encoding bit rate. Therefore, 
although, again, the invention is not limited in scope in this respect, for this embodiment, 
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adjusting the quantization step size is a mechanism employed to modify or adjust the video 
encoding rate. This follows at least in part from the observation that a high quantization step 
size provides relatively coarse quantization. Thus, the amount of information to be sent to the 
decoder is reduced when employing a high quantization step size. 



It may in this context be desirable to appropriately characterize the relationship between 
the bit count employed to encode a macroblock and the SAD of the macroblock, at least for 
this particular embodiment. Therefore, for different values of a quantization step size 
parameter, here from one to 31 , these particular parameters are computed for a variety of 
images. Of course, this is just one potential methodology and any one of a number of 
methodologies may be employed. The invention is not limited in scope to employing any 
particular methodology. Therefore, furthermore, in this particular embodiment, as shall be 
described in more detail hereinafter, the macroblocks (MBs) are classified by type, such as 
inter, intra, B and 4 MV. In this context, 'intra' refers to a MB coded without motion vectors. 
Inter* refers to a MB that uses one fonward motion vector, '4 MV refers to a MB that uses four 
foHA/ard motion vectors, and B refers to a MB that uses fonA/ard and backward motion vectors 
to reduce temporal redundancy, although, again, the invention is not limited in scope in this 
respect. It is noted that the modes also provide information based on motion estimation that 
may be useful in video encoding rate control. 



In this particular embodiment, although, again the invention is not limited in scope to 
employing this particular methodology, the SAD is obtained after motion estimation has been 
performed, such as at the point shown in the block diagram illustrated in FIG. 1 , except for 
intra macroblocks, of course. This point in FIG. 1 is chosen so that the mode of each 
macroblock using the results of motion estimation may be obtained for this particular 



embodiment. 
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Using this methodology or approach, a relationship between macroblock SAD and the 
count may be generated for each quantization parameter or step-size. In this particular 
implementation, based upon the quantization parameter or step-size, 31 figures may, 
therefore, be generated, although this is not intended to be limiting on alternate approaches 
within the scope of the present invention. Here, then, for each different quantization step-size, 
from a number of macroblocks having SADs, the total number of bits is determined. Likewise, 
as previously described, the different macroblock types may also be employed. The 
relationship between SAD and bit count may be shown to depend at least in part on type of 
macroblock, in addition to depending at least in part on the SAD of the macroblock, at least for 
this particular implementation; however, as previously indicated, the invention is not limited in 
scope to this particular implementation. 

In order to make the data generated suitable for use in video encoding rate control, it is 
desirable to quantize the macroblock SAD, although, again, of course, the invention is not 
limited in scope in this respect. For example, some other embodiments may employ the 
foregoing approach regarding SAD without applying quantization. Nonetheless, the following 
quantization technique is employed in this particular embodiment. Of course, any one of a 
number of other suitable techniques may alternatively be employed, and all such other 
quantization techniques are included within the scope of the present invention because the 
particular technique applied is not significant. However, in this particular embodiment or 
methodology, the following quantization technique is employed. 

index = SAD/bin_size 
8 
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[2] 



where 



bin_size = range/no_bins 



In equations [2], *SAD' is, of course, the macroblock SAD. Likewise, 'no_bins' is 8. For a 
given quantization step-size, in this embodiment, it is then desirable to average the bit count 
depending upon the particular index. This is illustrated by the plots in FIG. 2 depending on the 
picture types employed, in this implementation, types I, P, or B, It is noted that such picture 
types are employed in connection with MPEG compliant video encoders, although, again, the 
invention is not limited in scope to MPEG or compliance with MPEG. 

The plots shown in FIG. 2 illustrate on one graph the relationship between SAD and bit 
count where quantization step-size is held constant for each separate curve, but varied across 
the family of curves shown on each respective plot. Using this data, therefore, for a video 
encoder, the video bit rate to be employed may be varied to take into account the SAD of a 
macroblock, and/or the macroblock type. More specifically, by quantizing the total number of 
bits and the macroblock SAD, the plots as shown in FIG. 2 may be converted into lookup 
tables (LUTs) that may be stored and employed by a video encoder during the process of 
encoding video to apply video encoding rate control. For these plots, 40 bins were employed 
for bit count for each picture type, although, of course, the invention is not limited in scope in 
this respect. 

For such an embodiment, the following methodology may be employed, although, 
again, the invention is not limited in scope in this respect. For an image or frame, a maximum 
acceptable quantization step-size is employed. Then, using the index, here quantized SAD, 



)-size. from the look- up tables the rate, or aite 
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and the quantization step-sTze, from the look- up tables the rate, or alternatively the bit count, 
is computed for the frame or image. Here, this is done on a macroblock basis and then 
summed over the image or frame, although, of course, the invention is not limited in scope in 
this respect. If the rate or bit count that is computed for the frame is less than the rate 
indicated by the bit budget, this indicates that the quantization step-size may be lowered to 
provide better performance without exceeding the budget. Therefore, the quantization step is 
lowered and the previously described process is repeated until the bit budget or rate is 
exceeded, indicating that the quantization step-size reduction has reached a limit. 



In this particular embodiment, the maximum acceptable step-size is 20 for I and P type 
pictures and 28 for B type pictures. Therefore, for this embodiment, the following pseudo- 
code may be employed to implement this embodiment. 



For ( 1 =Qp_max; 1>= Qp_min; 1--) { 
sum_rate = 0; 
for (1=0; i<N; i-h-h) { 

for (j=0; j<M; j-h-h) { 

suin_rate += RATE [index] [1] ; 

} 

} 

if ( suin_rate>budget ) break; 

} 

Qp = 1; /* your desired Qp for current frame 

N, M are image height and width divided by 16. NxM represents number of macroblocks 
per frame. Sum__rate represents the estimated bits spent using the index and Qp 
relationship. Budget is the allocated bits for the current frame. 



In comparison with state of the art approaches to adjusting the video bit rate, an 
embodiment in accordance with the present invention has several advantages. For example, 
an approach referred to as Q2 is used in connection with MPEG-4. The target bit rate is 
computed based on the bits available and the last encoded frame. If the last frame is complex 
and uses excessive bits, more bits might be assigned to this frame. However, if there are 
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fewer bits left for encodingTrewer bits might be assigned due to the oit budget. Therefore, a 
weighted average provides a compromise between these two factors. 

Once the target bit rate for the frame is determined, the quantization step-size to meet it 
is selected. This is accomplished using a least squares statistical modeling technique. The 
encoder rate distortion function is modeled as: 



R = Xr--^X^~ [3] 



The encoding bit count is denoted R. The encoding complexity, denoted S, is measured using 
the mean absolute difference (MAD). The quantization step-size is the parameter Q. The 
modeling parameters, X, and X^, are estimated using least squares from previous data. Then 
the equation above is solved for Q. To solve the equation using this technique, typically up to 
20 previous frames of data are employed, suggesting computation complexity as well as 
employing significant memory. Furthermore, simulation results indicate that the Q2 technique 
does not meet the bit budget, that is, comply with the target rate, for all images. 

One advantage of this particular embodiment, therefore, is reduced computational 
complexity. For this particular embodiment, for example, one parameter, bit budget or rate is 
employed. Likewise, in this embodiment, although again the invention is not limited in scope 
in this respect, is macroblock SAD is employed. In terms of computational complexity, this 
does not produce a significant amount of additional overhead because the SAD is calculated 
to determine macroblock mode, as previously described, except for I frames. Furthermore, 
this computation for I frames, although providing some additional overhead, is not significant in 



11 



esources that are consumed. Likewise, astn€ 



042390.P10587 



terms of the processing resources that are consumed. Likewise, asthe results below indicate, 
the bit budget is met for all images. 



FIG. 3 is a table providing a comparison between various performance parameters for 
an embodiment in accordance with the invention and Q2. This data was generated from six 
image sequences. 150 frames from each was employed with a frame rate of 15 frame per 
second. The number of B frames between P or I frames is 2 and the intra period is 15 frames. 
The data in the table implies that the degradation in performance quality is slight, and in some 
cases is better. Furthermore, this embodiment stayed within budget with greater compression 
efficiency. 

The previously described embodiments provide a number of desirable advantages and 
features. For example, as previously explained, the implementation of a rate control 
mechanism, such as those previously described, reduces computation complexity. Therefore, 
although results may vary depending on a variety of factors, such embodiments may be 
suitable for low-power applications, as is often desirable. Likewise, the previously described 
embodiments may be implemented in hardware, software, firmware, or any combination 
thereof. Furthermore, embodiments in accordance with the invention provide compatibility 
with known video standards, such as MPEG and H.26x. 



It will, of course, be understood that, although particular embodiments have just been 
described, the invention is not limited in scope to a particular embodiment or implementation. 
For example, one embodiment may be in hardware, whereas another embodiment may be in 
software. Likewise, an embodiment may be in firmware, or any combination of hardware, 
software, or firmware, for example. Likewise, although the invention is not limited in scope in 
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this respect, one embodiment may comprise an article, such as a storage medium. Such a 
storage medium, such as, for example, a CD-ROM, or a disk, may have stored thereon a look 
up table, such as previously described. Likewise, a storage medium may have stored 
instructions, which when executed by a system, such as a computer system or platform, or an 
imaging system, for example, may result in an embodiment of a method in accordance with 
the present invention being executed, such as an embodiment of a method of performing 
video encoding rate control using bit budget, for example, as previously described. For 
example, a video processing platform or an imaging system may include a video encoder, a 
video input device and memory. The video encoder may include a mechanism to adjust the 
video encoding rate employed during video encoding, such as by employing one of the 
embodiments previously described, for example. Furthermore, embodiments of the invention 
are also not limited to video encoders or video encoding. For example, video may be decoded 
where the video had been encoded using an embodiment in accordance with the invention, 
again, such as previously described. 



While certain features of the invention have been illustrated and described herein, 
many modifications, substitutions, changes and equivalents will now occur to those skilled in 
the art. It is, therefore, to be understood that the appended claims are intended to cover all 
such modifications and changes as fall within the true spirit of the invention. 
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